@sebastian-nagel the test was left failing on purpose, because I wanted to first fix the test running.

I managed (or, better, Claude Opus managed - other models weren't able) to find the cause.
In short, all test were ran with <fork> and there were two tests in particular TestCommonCrawlDataDumper whcih called at some point System.exit() dumping the whole forked process. However also TestMimeUtil had a simliar problem.

For now I've separated the org.commoncrawl.* from the org.apache.* tests. But the issue may still occur preventing other tests from the same group to be executed.

called at some point System.exit()

Thanks! That's a left-over of NUTCH-2852. Unclear why it hits our fork but not upstream Nutch. It should be fixed upstream anyway. I'll also comment on PR #44.

@sebastian-nagel I believe now the test works fine with the data from a real case. 🌮 🎉
I think there are only two item that I did not find:

String payloadDigest = "sha1:abc123";
String blockDigest = "sha1:def456";

The digests are calculated one level higher in the WarcRecordWriter and then passed to the WarcWriter. No need to test it as part of this PR.